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Abstract 

This paper analyzes the current proposals by the government of Alberta, Canada 
to implement an accountability framework for the province's postsecondary institutions 
using performance indicators. The paper develops a conceptual framework for 
performance indicators based on a discrepancy model of evaluation using three 
metaphors: mechanical, medical, and economic. This is followed by a critical 
examination of performance indicators with a delineation of potential weaknesses and 
strengths as well as recommendations for practice. Finally, the Alberta plan is used as 
a case study to illustrate the concepts developed in the paper. 

The paper argues that the Alberta plan for performance indicators in the public 
postsecondary sector reflects an economic metaphor of /^rformance indicators which 
will result in measuring fiscal effectiveness as opposed to educational effectiveness. 
Further, the paper argues, while the Alberta government has indicated that the 
performance indicators will allow for inter- and intra-sectoral variations, no allowances 
seem to have been made for a value-added assessment of student outcomes which is at 
the heart of the purpose for postsecondary educational institutions. 
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In March 1994 the Alberta Department of Advanced Education and Career 
Development produced its first Three-year Business Plan (1994b). At the same time the 
department was in the final stages of completing a review of its mission and mandate 
which included significant public consultation and input. New Directions for Adult 
Learning in Alberta (Alberta Advanced Education and Career Development, 1994a; 
hereafter referred to as New Directions) , the policy paper which resulted from that 
review, was published in October 1994. These two documents emphasized 
accountability. This thrust was in keeping with government actions to develop and 
publish performance indicators (Pis) in all government departments. The government 
has consistently trumpeted its use of a business approach to governing. 

New Directions outlined four goals which the department set for itself with 
respect to Alberta's postsecondary education sector. The goals were to increase 
accessibility, responsiveness, affordability, and accountability. Within the scope of the 
accountability goal the policy paper indicated the government intends to "require 
[education] providers to measure and report on performance through an accountability 
framework to advise Albertans of results achieved in publicly funded learning 
opportunities" (p. 16). Reporting, the policy paper said, is to be done by Pis. 

Since New Directions was published the department of Advanced Education 
and Career Development has been working with the postsecondary sector to develop 
these Pis. As has often been the case in other jurisdictions, the process has been long 
and difficult. The initial plan was to have Pis implemented in 1995/96 (Alberta 
Advanced Education, 1994b); but 1995/96 has become a year for piloting and 
implementation has been put off for another year. However, the government has 
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shown the political will to proceed with implementation. A key question 
postsecondary institutions in Alberta had to answer was the degree to which they 
would cooperate so as to have input into the creation of the accountability framework. 
Given the track record of the government in reforming the health and education 
sectors, if the postsecondary institutions had not participated, it is likely that the 
government would have proceeded to impose a set of Pis— some of which might not 
have been in the best interests of some institutions. 

Why the flurry of activity to develop Pis and implement them within so quickly? 
What are Pis and are they applicable to Alberta's postsecondary system? This paper 
explores these questions by presenting a conceptual framework for Pis and examining 
the nature of the proposed Alberta PI system in the light of the framework. The 
conceptual framework uses three metaphors to explain the basis for Pis. 

The Alberta Context 

It is helpful to provide context for this paper by briefly describing the 
postsecondary system in Alberta. Alberta has a population of 2.4 million. Its citizens 
are served by four universities, 11 community colleges, two institutes of technology, 
four government administered vocational colleges, and the Banff Centre for Continuing 
Education. Four community education consortia, 85 community adult learning 
councils, a wide range of private colleges, and more than 90 licensed private vocational 
schools complement the traditional educational institutions (Andrews & Elford, 1996). 
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Conceptual Basis for Performance Indicators 
Pis in higher education developed as an outgrowth of the outcomes assessment 
movement of the last decade. Ewell (1983) defined student outcomes "as any change or 
consequence occurring as a result of enrollment in a particular educational institution 
and involvement in its programs"(p. 11). Easley (1987) further clarified what is meant 
by student outcomes assessment by defining assessment as "the process of locating 
evidence to show that specific or general outcomes have been achieved by the students" 
(p. 10). Ewell also pointed out that student outcomes assessment as developed in the 
last 30 years "views the assessment of outcomes as part of the process of rational 
resource allocation and program decision making" (p. 4). Outcomes assessment and, 
thus. Pis are rooted in a rational systems approach to organizations. 

According to the systems model institutions ought to develop goals that 
determine the outcome for which they strive. It then follows that organizations should 
evaluate the degree to which they attain those goals. Outcomes assessment techniques 
are used to provide information on the degree of congruence between intended student 
outcomes and actual student outcomes. This information in turn provides feedback to 
guide the curricular decisions. This process, which is similar to Provus' (1971) 
discrepancy evaluation model, is outlined in Figure 1. 

The concept of Pis can been illustrated using mechanical, medical, and economic 
metaphors. In education Pis can be understood as attempting to demonstrate the 
effectiveness and efficiency of an institution in attaining its goals similar to the way 
gauges and lights are used show the efficient or effective operation of a motor. 
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instruments the health of a person's body, or statistics the state of an economic 
enterprise be that a company or a nation. 




Figure 1. Relationship between Intended Student Outcomes, Student Outcomes, and 
Curricular » lanning 

Mechanical Metaphor 

Nuttall (1994) argued that educational Pis are like mechanical monitors. His 
emphasis was on the use of feedback to monitor efficiency and effectiveness. Parallels 
can be drawn to educational Pis by considering the kind of monitors we use for an 
automobile. Direct observation of the mechanical aspects of a car is not possible while 
driving. Therefore, automotive engineers have designed instruments which measure 
certain characteristics and provide proxy indicators of its performance. Immediate 
feedback is important to monitor some features of a car's performance. Other Pis can 
be checked at certain intervals. However, both kinds of monitors are, in effect, 
formative evaluation tools which provide information to guide decisions about the 
need for closer examination. As Nutlall said, 
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they act as an early-warning system that something may be going wrong, in the 
same way that the instruments on the dashboard of a car can alert the driver to a 
problem or reassure him or her that everything is functioning smoothly. A dial 
pointer moving into the red zone is only a symptom of some malfunction and 
further investigation is needed to establish the cause, (p. 17) 

Pis in automobiles include such things as the rate or fuel consumption, engine 
temperature, brake line pressure, and tire wear patterns. From these Pis we can infer 
something about past, present, or future performance. Nuttall (1994) also pointed out 
that "if something is wrong, the indicators themselves do not provide the diagnosis or 
prescribe the remedy; they are simply suggestive of the need for action" (p. 17). In a 
similar fashion., one intended use of educational Pis is to provide information which 
may give an indication of the efficiency or effectiveness of an educational system and 
alert people if there is the need for further attention. 

Medical Metaphor 

The medical diagnosis metaphor is also helpful in understanding Pis. Although 
direct examination is possible most medical practitioners try to be as non-invasive as 
possible because of financial costs as well as personal costs in time and comfort. 
Invasive surgical examinations for diagnostic purposes is inefficient. Instead, most 
diagnoses are made by proxy indicators such as temperature, white blood cell counts, 
or blood pressure. 

The medical metaphor also provides insight into the use of Pis in education. 
Proxy indicators are used because of the difficulty, disruption, complexity, and cost of 
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direct examination. Furthermore, the idiosyncratic nature of those analyses would 
make inter-institutional comparison difficult if not impossible. As in medicine, 
norms— including ranges of acceptable deviation— as well as standard definitions and 
procedures are needed so that measures are comparable across institutions and time. 

Economic Metaphor 

Nedwek and Neal (1994) noted that many authors see a natural parallel between 
economic indicators and educational Pis. Economists generally describe the economic 
state of an organization or nation by reference to a few statistics. Cave, Hanney, and 
Kogan (1991) pointed out that each statistic is based on numerous components but is 
generally expressed in some quantitative measure. We are used to such national 
economic Pis as gross domestic product, consume! price index, stock market indices, 
balance of trade, government budget deficits, and unemployment rates. Those 
interested in corporate health look to corporate economic Pis such as unit cost of 
production, profit margin, cash flow, or debt to equity ratio. 

As with the mechanical metaphor, economic Pis generally identify changes in 
effectiveness or efficiency. Nedwek and Neal (1994) observed that the economic PI 
approach has much support in higher educational institutions because of the frequent 
presence of corporate leaders on governing boards. To the extent that this same 
tendency is true in government circles, we might expect the economic metaphor to be 
applied to public postsecondary education institutions. 

Medical and mechanical Pis tend to be criterion- or norm-referenced. Criterion- 
referenced systems have specific, predetermined targets against which performance is 
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judged. Norm-referenced systems use average performance of a larger group as the 
standard of performance. However, in economic PI systems the point of reference for 
interpretation tends to be past performance. Thus, economic Pis tend to be expressed 
as percentage or absolute value changes from some previous period (typically «i quarter 
or year). In medicine and mechanics trends but cease to be important if the PI is within 
the normal or target range. It would be rare indeed to get ecor jmists to agree on what 
the normal range of unemployment or inflation should be. 

Educational PI systems tend to possess aspects of all three metaphors. 
Enrollments, for example, are generally historically-referenced (e.g., change from the 
previous year); however, when expressed another way (e.g., student-faculty ratio) they 
may have a norm-reference or even criterion-reference (e.g., government funding 
targets) interpretation. 

Windham (1990, as cited in Nuttall, 1994) drew attention to the fact that 
economic Pis are required to be "accurate, relevant, timely, understandable and 
affordable" (p. 34). Those who support the economic approach to Pis would likely 
agree to use such criteria in evaluating the adequacy and appropriateness of 
educational Pis. In order for Pis to be implemented and used to guide institutional 
policy decisions, such characteristics would appear to be appropriate. 

.Weaknesses of Performance Indicators 

There are several potential weaknesses of Pis with respect to their use in 
postsecondarv education which need to be guarded against both in Alberta and as the 
Alberta model may be applied in other jurisdictions. These are discussed below. 
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Economic Metaphor Bias 

There is a tendency to apply only a single kind of Pis. In many cases, as 
Nedwek and Neal (1994) pointed out, the single metaphor used is the economic one 
which results in institutions measuring only fiscal health. They argued that most of the 
language relating to the design of PI systems has been drawn from industry, yet the 
nature of higher education, “with its dual system of control between administrative 
and professional cultures" is not sufficiently similar to allow for "direct application of 
the industrial input/ output model" (p. 81). Astin (1991) argued convincingly that the 
true measure of an educational institutions effectiveness is its impact on the lives of 
students. In addition, universities have a second major function —research. An 
appropriate set of Pis for universities must include defensible Pis for research. 

Sector Tensions 

A system of Pis must allow for inter- and intra-sector variations. Inter-sector 
variations are those between different types of institutions. A system with universities, 
colleges, vocational and technical institutes, and private institutions, as in Alberta, 
presents different demands on a PI system than is the case for a uniform system. Too 
strong a concern with system-wide comparability of Pis can result in Pis which indicate 
nothing but similarities and bury the distinctiveness of each sector. On the other hand, 
a system which is premised on the uniqueness of each institution and each program 
may yield Pis which are so discrete that comparison is impossible. 

Intra-sector variations are between different institutions within the same sector. 
Within its university sector, Alberta has one small regional university, a university 
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based on a distance-delivery model, and two large graduate research universities. 

Other jurisdictions may have differences resulting from program specialization or 
geographical variations. Similar challenges face the development of intra-sectoral PI 
systems as those identified for inter-sectoral systems. The difficulties involved in 
developing PI systems which can balance the inter-sectoral and intra-sectoral 
differences are considerable. 

Variations in Interpretation 

A third potential weakness of Pis is reflected in the argument for differentiated 
Pis. Interpretation of the meaning of a PI is subject to variations in expectations, 
worldviews, and interests. Different stakeholders may respond in opposite ways to the 
same PI depending on what they consider important. For example, students who want 
a close relationship with instructors might respond quite negatively to a student-faculty 
ratio of 50 to 1, whereas a provincial treasurer concerned with the cost of postsecondary 
education may say such a ratio is too low considering the infrastructure costs of large 
research university. The basis for interpretation of Pis is needs to be settled before 
implementing a system of Pis. 

The criteria for evaluating student learning Pis must be clearly established 
because that is one of the key goals of postsecondary institutions. Halpern (1987) 
argued student outcomes should only be evaluated with respect to intended outcomes. 
A complicating factor is whether the intended outcomes to be considered should be 
those of the institution or the student. 
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Political expectations must be considered. Governments may want some Pis 
developed because of their political value rather than their ability to provide feedback 
on goal attainment by institutions or students. Governments seeking to reduce public 
expenditures may use the bottom line as the criterion for nearly everything they do, but 
providing a quality education is not the same as providing a quality education at the 
lowest possible cost— no matter how quality is defined. If the Pis focus on economics 
rather than academics, resulting actions will focus on the fiscal rather than the 
educational. 

Over-Quantification 

Nuttall (1994) observed that while there is disagreement on the definition of Pis, 
"it would seem that the more common view of indicators is of the quantitative variety" 
(p. 18). He pointed out that some writers limit the meaning to a narrow quantitative 
one as did Johnstone (1981) in saying "an indicator is . . . something which is 
quantifiable" (p. 4). Shavelson, McDonnell, Oakes, Carey, and Picus (1987) defined an 
indicator as "an individual or a composite statistic that relates to a basic construct in 
education and is useful in a policy context" (p. 5). However, they distinguished 
between statistics and indicators by noting that "statistics qualify as indicators only if 
they serve as yardsticks" to measure educational quality (p. 5). 

Yet, many aspects of education goals resist quantification. Educational aims and 
outcomes are diffuse. How do we determine, and even if that is possible how do we 
express, in a meaningful way using numbers that students have grown in such aspects 
as civic responsibility as a result of the educational experiences they have had in an 
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institution? Yet, politicians and members of the public and the media who demand 
accountability seem to also be demanding quantification. The challenge is to provide 
meaningful information to those stakeholders while also generating insightful 
evaluative information for policy makers and practitioners. 

Strengths of Performance Indicators 

Nedwek and Neal (1994) identified these four positive benefits of PI systems: 

(a) monitoring the broad context of a policy, (b) providing benchmarks with 
relationship to specific goals, (c) predicting and/ or providing early detection of 
emerging problems, and (d) providing explanations for existing problems. An 
additional benefit is the opportunity for enhanced rationality of decision-making. 

Context Monitoring 

In Total Quality Management terms. Pis can provide information which enable 
stakeholders to monitor the environment in the same way as economic Pis do for 
business. In the current competitive environment educational institutions need to have 
a means of tracking trends both internally within their various departments and 
externally in society and other comparable institutions. 

Goal Attainment Indicator 

Cave, et al. (1991) argued Pis provide information with respect to the degree to 
which desired outcomes have been achieved. Jacobi, Astin, and Ayala (1987) 
maintained outcomes assessment can be used at four critical points of the strategic 
planning process: (a) goal determination, (b) process refinement, (c) baseline data 
development, and (d) feedback on goal attainment. Similar to their use in the medical 
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and mechanical metaphors. Pis can summarize results from student outcomes 
assessment activities and provide information concerning goal attainment so that 
follow-up plans and actions can be implemented at the appropriate time. 

Early Problem Detection 

When Pis are used as process monitors they provide the opportunity to detect 
problems before they get out of control. The mechanical metaphor illustrates the value 
of Pis in providing early warning of variance from the accepted norms or standards. 
However, this requires clear delineation of the anticipated performance at critical 
stages of the process. 

Problem Explanation 

Problem explanation requires a different kind of information from what is 
needed for problem identification as illustrated in the mechanical metaphor. To the 
extent the medical diagnostic metaphor can be applied in education. Pis can provide 
information about aspects of a system which are performing abnormally such that 
understanding of the problem is possible. The key to such use rests in the development 
of a comprehensive set of norms for educational performance. Measurements and 
reports must be sufficiently detailed to provide the kind of information that will enable 
managers to identify problem components is a system. A government imposed PI 
system for a multi-sector postsecondary system would be unlikely to have the 
sophistication necessary to provide this kind of input. Such a system requires 
designers with specific experience and knowledge of the institutions and programs. 
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Enhancement of Rational Decision-Making 

Pis attempt to increase the rationality of curricular decision-making by 

providing information regarding the functioning of the instructional/ curricular 
aspects of an institution. Writers vary in the degree to which decision-making is 
considered a rational process. At one end are those who support a rational approach 
like Saupe (1981) while at the other end are those like Cohen, March, and Olsen (1972) 
with their Garbage Can model of institutional decision-making. In between we can 
place Simon's (1993) bounded rationality model as well as political models and the 
various models emphasizing participation. Following Estler' s (1988) categorization, 
decision-making models can be placed on a continuum (see Figure 2) with respect to 
the degree of rationality inherent in the process. 

Rational-bureaucratic 4 ► Irrational/chance 

Ideal Weberian Simon's Bounded Political and Garbage Can 

bureaucracy Rationality model participatory models* model 

*These models vary by the level at which the rationality occurs — group vs. Individual 

Figure 2. Degree of Rationality of Decision-making Models 

Rational decision-making is bounded, in Simon's terms, by several factors, but 

inadequate or inappropriate information is critical. It would be unreasonable to 

suggest that we will, or should, ever approach the Weberian ideal; however, 

appropriate and timely information can do much to assist decision-making. Providing 

appropriate and timely information is proposed as a fundamental function of Pis. 

Abuse of Pis for political purposes should not be allowed to negate their usefulness. 
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The Alberta Plan 

History of Development 

Alberta Advanced Education and Career Development has been seeking to 
develop a system of Pis tor the postsecondary institutions since 1993. In the New 
Directions policy paper the department identified these three concerns with 
accountability practices as they existed in Alberta at that time: (a) no system-wide 
accountability approach with common definitions and comparability of information, (b) 
inadequate public access to information expenditures of tax money by postsecondary 
institutions, and (r) a lack of "current and objective information to assist [potential 
students] in making informed choices about opportunities to learn" (p. 16). In 1995 
Alberta Advanced Education and Career Development (1995) indicated it was also 
concerned that existing performance measures and reporting procedures did not 
provide information at the program level, particularly fiscal information. 

In December 1994 the presidents of the various public postsecondary institutions 
in Alberta agreed to 24 expected outcomes for which Pis would be developed (Larry 
Orton, personal communication, March 29, 1995). These outcomes relate to enrollment, 
transferability, program completion, student and graduate satisfaction, rates of 
employment in related occupations, continuation toward educational goals, employer 
satisfaction with graduates, receiving institution satisfaction with transfer students, 
community service, university research, fiscal management, and system 
responsiveness. Earlier efforts to develop common definitions for enrollment and 
funding purposes have been used as a basis upon which to reach common 
understandings regarding what Pis will be implemented. 
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Evaluation of Alberta's Proposed Performance Indicators 

A review of the 24 expected outcomes indicates the focus will be on economic 
concerns. Of the 24 expected outcomes for which Pis are being developed, six deal 
with fiscal issues and eight others use fiscal measures to provide indicators of 
performance. For example, in order to determine the degree to which community 
service is perceived to be satisfactory to the general public the proposed Pis intend to 
use economic impact studies. While an opinion poll might be more appropriate to 
determine public attitudes on this issue, it appears that the politically correct response 
may be more likely to occur. Many of the other proposed Pis relate to enrollment, 
acceptance, transfer, completion, utilization, and satisfaction rates. 

In addition to the four types of public postsecondary institutions in Alberta, 
there are various private postsecondary educational institutions which receive 
government funds and, therefore, are included within the scope of the accountability 
net which is to cover “publicly funded learning opportunities" (Alberta Advanced 
Education, 1994a, p. 16). There are also intra-sectoral distinctions. As has been argued 
above such differences necessitate different Pis. The department appears to have 
recognized this need. New Directions acknowledged ii i*it Athabasca University, as a 
distance learning provider, was unique and the department has acknowledged 
Athabasca University will not be measured by the same Pis as the other universities. 
The University of Lethbridge, predominantly an undergraduate institution, ought to be 
evaluated using different standards and means than are used for the two large research 
universities (University of Calgary and University of Alberta). 
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Within the college sector there are important differences between institutions on 
the basis of size, specialization, and location. For example, differences of accessibility 
and the impact of regional economic conditions ought to be considered when 
comparing institutional Pis measuring student demand. New Directions, appeared to 
recognize these differences when it stated that "comparable indicators of performance 
across sectors will be developed wherever possible, with the distinctive roles and 
objectives of each sector being respected" (p. 17). The February, 1995 progress report 
(Alberta Advanced Education, 1995) strengthened the commitment to differential 
interpretation of Pis. The progress report stated that 

information must be comparable across the system. Performance indicators 
must be relevant to the goals and objectives ot each institution. Institutions must 
be comparable on the particular performance indicator. For example, it would 
be inappropriate to compare the province's two technical institutes with smaller 
regional colleges, (p. 9) 

On the other hand, it does not appear that the government of Alberta intends to 
develop interpretive criteria prior to the implementation Pis. The February 1995 
progress report (Alberta Advanced Education, 1995) stated that benchmarks will be 
established using the data gathered in the first year of implementation. Establishing 
benchmarks in this way will use current practice as the standard. The interpretative 
framework being used is historical and norm-referenced rather than criteria-referenced. 
If all the institutions do an inadequate job in one area the benchmark will be set 
artificially low and will not provide the incentive for improvement that would be 

20 



needed. 



Performance Indicators in Alberta 19 



New Directions proposed that "Albertans will be provided with key indicators 
of program, service and overall performance (p. 12) and that these indicators . . . be 
published by providers in calendars, annual reports or business plans" (p. 16) in order 
to address the problem of inadequate information available to students. However, the 
issue of providing a context for the interpretation of the information appears to remain 
unresolved. Institutions may find themselves attracting the wrong students because of 
the enforced r porting if no interpretive context is provided. The degree to which 
performance conforms to the institutions goals needs to be communicated if the 
information is to be meaningful. Without the goals and outcomes being contrasted 
student dissatisfaction may result. That dissatisfaction will result in poorer PI results 
which will be regarded by government as poor performance. The institution ultimately 
gets penalized for attracting dissatisfied students, whom it did not want to attract but 
who came because of the enforced publication of information without that information 
being placed in the context of the institution's mission. 

Alberta Advanced Education and Career Development (1995) indicated Pis are 
to "provide Albertans with the information necessary to evaluate the overall success of 
the post-secondary sector in ensuring an accessible, responsive and affordable system 
of quality learning opportunities" (p. 10). The ultimate measure of quality for higher 
education is whether adul s are developing the skills, knowledge, and attitudes which 
they desire and need. In light of this purpose this intention is laudable. However, the 
rhetoric appears to be stronger than action. The proposed Pis for the 24 expected 
outcomes agreed to in December 1994 fall short of reflecting whether there is a system 
of quality learning opportunities" because Pis which might show the "value-added" (to 
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use Astin's, 1991, phrase) by the learning experiences of the students are not attempted. 
Satis faction of students, employers, and receiving institutions will give some indication 
of the perceived benefit of the learning experience, but these measures are inadequate 
to indicate the degree to which learning goals of students, employers, or the 
postsecondary institutions themselves are being met. 

Conclusion 

The mechanical, medical , and economic metaphors are helpful in analyzing the 
proposed PI system in Alberta. The government 7 s economic approach is reflected in 
the historically comparative nature of many of the proposed Pis for Alberta's 
postsecondary education system. 

Alberta's experience in developing a PI system for a multi-sector postsecondary 
education system reinforces the inherent difficulties in such an endeavor. The Alberta 
approach, despite (or, perhaps, because of) using a consultative development process, 
appears to be heavily weighted toward the economic metaphor. 

The government needs to be commended for recognizing the inherent problems 
with trying to develop a comprehensive PI system which will be fully transferable 
between and within sectors. However, the department does not seem to have realized 
the inability of an economic style PI system to reflect the true nature of institutional 
achievement in relation to educational goals. 




22 



Performance Indicators in Alberta 21 



References 

Alberta Advanced Education and Career Development. (1994a). New directions f or 
adult learning in Alberta . Edmonton, AB: Author. 

Alberta Advanced Education and Career Development. (1994b). Three-year business 
plan . Edmonton, AB: Author. 

Alberta Advanced Education and Career Development. (1995). Institutional 

accountability in Alberta's post-secondary system: A progress report . Edmonton, 
AB: Author. 

Andrews, M. B., & Elford, I. C. (1996, May). New directions for adult learning in 

Alberta: One year later . Paper presented at the annual conference of the Canadian 
Society for the Study of Higher Education, St. Catherines, ON. 

Astin, A. W. (1991). Assessment for excellence: The philosophy and practice of 

assessment and evaluation in higher education . New York: American Council on 

■ 5 

Education, Macmilla .. 

Cave, M., Hanney, S., & Kogan, M. (1991). The use of performance indicators in higher 
education: A critical analysis of developing practice (2nd ed.). London: Jessica 
Kingsley Publishers. 

Cohen, M. D., March, J. G., & Olsen, J. P. (1972). A garbage can model of 
organizational choice. Administrative Science Quarterly, 17 (1), 1-25. 

Easley, R. R. (1987). Assessment of student outcomes in the Bible college: 

Identification of important outcomes and current practices . Unpublished doctoral 
dissertation, University of Arkansas, Fayetteville. 




23 



Performance Indicators in Alberta 22 



Estler, S. E. (1988). Decision making. In N. J. Boyan (Ed.), Handbook of research on 
educational administration , (pp. 305-320). New York: Longman. 

Ewell, P. T. (1983). Information on student outcomes: How to get it and how to use it . 

Boulder, CO: National Center for Higher Education Management Systems. 
Halpern, D. F. (1987). Student outcomes assessment Introduction and overview . 

New Directions for Higher Education, 59, 5-8. 

Jacobi, M., Astin, A., & Ayala, F., Jr. (1987). College student outcomes assessment: A 
talent development perspective . ASHE-ERIC Higher Education Report No. 7. 
Washington, DC: Association for the Study of Higher Education, 

Johnstone, J. N. (1981). Indicators of education systems . London and Paris: Kogan 
Page and UNESCO. 

Nedwek, B. P., & Neal, J. E. (1994). Performance indicators and rational management 
tools: A comparative assessment of projects in North America and Europe . 
Research in Higher Education, 35(1), 75-103. 

Nuttall, D. L. (1994). Choosing indicators. In K. A. Riley & D. L. Nuttall (Eds.), 
Measuring quality: Education indicators-United Kingdom and international 
perspectives, (pp. 17-40). London: Falmer Press. 

Provus, M. (1971). Discrepancy Evaluation: For Educational Program Improvement 
and Assessment. Berkeley, CA: McCutchan. 

Saupe, J. L. (1981). The functions of institutional research. Tallahassee, FL: 

Association for Institutional Research . (ERIC Document Reproduction Service No. 
ED 207 443) 




2'i 



Performance Indicators in Albr 'ta 23 



Shavelson, R. J., McDonnell, L., Oakes, J., Carey, N., & Picus, L. (1987). Indicator 
systems for monitoring mathematics and science education . Santa Monica, CA: 
RAND Corporation. 

Simon, H. A. (1993). Decision making: Rational, nonrational, and irrational. 
Educational Administration Quarterly, 29(3), 392-41 1 . 



